P ( x) could be nonconvex.

Bayesian optimization has recently emerged as a popular method for the sample-efficient optimization of expensive black-box functions.

Why compare to COBYLA instead of BOBYQA?

We thank all the reviewers for their insightful and encouraging feedback. Due to the discrete nature of COMBO's search space, the implementation detail is slightly different. In contrast, in COMBO's combinatorial graphs, we have spray vertices. As R2 suggested, this heuristic promotes exploitation. Using random vertices for exploration is similar to Spearmint.

Bound of GEM gradient estimation error ( Section 3.2) We show a general proposition in VI (or other measure approximation methods). However, it's not easy to model an arbitrary prior distribution with effective and efficient Bayesian inference. For single cluster of tasks, we show empirical evidences in Appendix C that there exist such kind of a distribution. In this work we focus on the uni-modal situation and leave the multi-modal situation to future work. B.3. co-ordinate descent ( Section 3.2) Following the ELBO property mentioned in Section 3.2 we have max Line 4 of Subroutine GEM-BML and Line 10 of Algorithm 1. B.4. recasting related works to our framework ( Section 4) For simpicity, we first set up some notations as follows: sg: stop gradient D This tensor building and backProps procedure has several drawbacks.